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ABSTRACT 



The National Reporting System project was designed to 
establish an outcome-based accountability system for the state-administered, 
federally funded adult education program. In the pilot test, volunteer states 
and their local programs compared the feasibility and costs of these three 
data collection models: (1) local programs collect all the student 

information and outcome measures, including measures by phone; (2) states 
conduct the follow-up measure survey by phone and then combine that data with 
other student data collected by local programs; and (3) state employment and 
human services agencies and higher education institutions combine their data 
with that from the state department of education. Findings for Model 1 (eight 
programs) indicated survey costs ranged from $500 to over $1500; the largest 
concerns were cost and staff availability; and programs reported general 
satisfaction with the educational functional level measures . The primary 
advantage was having the sampling universe at the program rather than state 
level. Disadvantages were high costs and time consumption. Findings for Model 
2 (three states) indicated costs varied from $40.37 to $132.04 (significantly 
less than the local survey) , and it alleviated the burden of data collection 
from local programs . The biggest drawback was that outcomes were not 
identified at the program level. Findings for Model 3 (four states) indicated 
barriers were confidentiality issues and development of measure definitions 
common across agencies; vastly different costs for developing and maintaining 
the database system; and a considerable investment in training and technical 
assistance at the local level was required to ensure valid, reliable data. 
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I. Overview of the NRS and Pilot Test 



The National Reporting System project (NRS) is developing an outcome-based accountability 
system for the state -administered, Federally funded adult education program. The NRS continues a 
cooperative process, begun more than five years ago, by the Division of Adult Education and Literacy 
(DAEL) of the U.S. Department of Education, state adult education directors and other adult education 
stakeholders, to identify, define and measure appropriate learner outcomes for the program. 

The NRS is envisioned as including a common set of descriptive, process and outcome measures 
with a standardized data collection methodology. Local programs will collect and aggregate these data 
from students at each instructional site, using an individual student record system. These data will be 
submitted to the state for aggregation and then submission to DAEL in aggregate form. DAEL will 
combine states' data into a national database describing participants and outcomes of adult education. 

Both states and DAEL can use NRS data to improve the public accountability of the adult education 
program, to assess progress in meeting programmatic goals and to relate effective practices and programs 
with successful outcomes. 

The proposed voluntary nature of the NRS changed in August 1998, when the Adult Education 
and Family Literacy Act within the Workforce Investment Act (P.L. 105-220) became law. This Act 
established accountability requirements, including that states develop outcome-based performance 
standards for adult education programs as one means of determining program effectiveness. The NRS 
mandate was then expanded to establish the measures and methods to conform to the Workforce 
Investment Act requirements. 

NRS Project Activities 

The NRS will establish a national accountability system for adult education programs by 
identifying measures for national reporting and their definitions, establishing methodologies for data 
collection, developing software standards for reporting to the U.S. Department of Education and 
developing training materials and activities on NRS requirements and procedures. The project is designed 
to conduct these activities in three phases. 

The first phase, standardization, involved the development of standard measure definitions for 
state and local programs, standard data collection methodologies, and software standards for automated 
data reporting. This phase was completed in the summer of 1998, when interim software standards were 
established, methodologies were identified for pilot testing and draft definitions for use in the pilot test were 
distributed to state directors of adult education. 

The pilot test, the subject of this report, was the second phase of the project and was designed to 
have a small number of volunteer states and local programs test the draft measure definitions and 
proposed methodologies under realistic conditions. The pilots assessed whether the draft measure 
definitions work or need refinement, as well as the costs, burden, and other difficulties in collecting the 
data using the proposed methodologies. 

Following the pilot tests, Phase 3, training and technical assistance, will be provided to state and 
local programs to support implementation of the NRS. The different types of assistance will include: 
instructional training packets that will be suitable for states to use in a ’’train the trainer” environment; 



ERIC 



4 



technology-based materials for state and local staff that explain the NRS measures and methods; and 
individual technical assistance to states to support their implementation efforts. 

NRS Measures and Pilot Tests 

Three types of measures are proposed for the NRS: 

• Student descriptive measures, including demographics and goals for attending; 

• Student participation measures, such as contact hour received. 

• Student outcome measures in four areas: learning gains within education functioning levels, 
advancement to further education and training, credentials obtained and employment measures 
(obtained and retained employment). Optional measures are identified in other areas, such as 
family literacy (such as reading more to children), and community (such as voting and 
community involvement). 

Table 1 summarizes the NRS measures used in the pilots. A separate document. Measure Definitions 
for the National Reporting System for Adult Education describes the measures used by the pilots in 
greater detail. 




TABLE 1 

Summary of Pilot Test Measures and Definitions 

Measures Categories or Definitions 



• Ethnidty 



^haXiol. onrJ. 6-oa.U 













r-'' 

■ ■■■ 



• Labor force status 



• Public assistance status 

• Disability status 

• Program enrollment type 



Learner goals for attending 



• American Indian, Asian, Black, 
Hispanic, White non-Hispanic 

• Male, female 



Employed, unemployed, not in labor 
force 

Receiving or not receiving assistance 
Disabled, not disabled 
Family literacy, workplace program, 
homeless program, correctional facilities, 
community corrections programs, other 
institutional program 

Obtain a job, retain current job, improve 
current job, obtain a high school diploma 
or GED, advance to post-secondary 
education or job training, improve ability 
to communicate with others, citizenship, 
mandatory, other personal goal 



best copy available 
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TABLE 1 

Summary of Pilot Test Measures and DefinKions 
Topic Measures Categories or Definitions 


.. ■ 
fc.' 


• Contact hours 

• Reasons for not completing planned 
service 

• Educational gains 

• Citizenship skills 


• N umber of hours of instructional activity 

• Illness/incapacity, lack of child care, lack 
of transportation, family problems, time or 
location of services not feasible, lack of 
interest instruction not helpful, moved, 
entered employment entered other 
education or training program, other 
known reason, cannot locate or contact 

• Six educational functioning levels each 
for ABE and ESL in reading, writing, 
speaking, and listening; and functional 
areas 

• Learners who achieve the skills needed 
to pass the citizenship exam 


tcohoivuc 
^ 
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• Entered employment 

• Improved employment 

• Retained employment 

• Reduction in receipt of public assistance 


• Learners who obtain a job while receiving 
instruction or within six months of ending 
instruction 

• Learners who obtain a new job with 
increased pay or benefits, more 
responsibility, or requiring higher skill 
levels while receiving instruction or within 
six months of ending instruction 

• Learners who remain in their job at least 
six months after exit from the program 
due to upgrading their skills 

• Students whose TANF or equivalent 
public assistance grant is reduced or 
eliminated due to employment or 
increased income 


- -■■ 

C4£jehi?«Xs fictCcu-neA g 


• Obtained a GED or High School Diploma 


• Number of learners who obtain a GED, 
high school diploma or recognized 
equivalent 


. 5AuCa^ti^$y^ OY^ I'Uunm^ ^ 
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• Advancement to further education or 
training 


• Learners enrolling in an educational or 
occupational skills program building on 
prior services or training received 



Under the NRS, local programs will be required to collect the descriptive and participation 
measures, along with measures of learning gains, while students are enrolled in the program. The 
remaining outcome measures assess the program effects on employment and further education and must 
be collected at some time after the student leaves the program. Thus, a follow-up methodology is required 
for states and local programs to collect these measures. 

The following three methodologies were imder consideration for the NRS. 

1. Local program reporting. The local program reporting model has local programs collect all 
student information and outcome measures, including the follow-up measures. Local 
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programs collect these longer-term outcome measures through a phone survey of a sample of 
learners who left the program six months previously. 

2. State follow-up survey. In the state follow-up survey, the state collects the longer-term 
outcome measures through a phone survey of a sample of students statewide. These data are 
then combined at the state level with the other student data collected by local programs. 

3. Data matching. Under this methodology, state employment and human service agencies, and 
often institutions of higher education, combine their data with the state department of 
education to create a shared, interagency database. Each agency then uses the shared data 
base to identify outcomes for their participants by matching records with Social Security 
numbers. For example, unemployment insurance (UI) records can provide data on wages and 
employment. 

The main purpose of the pilot test was to compare the feasibility and costs of these three data 
collection methods for use in the NRS. The experiences of the pilot states and programs were to inform 
DAEL on how the NRS should operate on a national level, including the follow-up methodology to employ 
and to identify measures that should be included or dropped and redefined. 

Eight local programs in as many states volunteered to participate in the local program pilot by 
conducting the follow-up survey on learners who left their programs sbc months earlier. The local 
programs also reviewed the other proposed NRS measures to provide feedback on the clarity and usability 
of measure definitions. In particular, the local programs reviewed the proposed educational functioning 
level descriptors to assess how well their staff could use them in placing and assessing learners. 

Three states volunteered to conduct the state level survey on a statewide sample. Data matching 
was assessed through case studies of four states that currently are doing data matching. Through these 
studies the key issues and approaches were identified. 

Overview of this Report 

This report summarizes the NRS pilot tests and their findings and discusses the implications for the 
NRS. The next section of this report presents an overview of, and findings from, the local program pilot. 
Section III describes the state pilot, the procedures employed for conducting a state -level follow-up 
survey, as well as advantages and disadvantages to employing this specific methodology. Section IV 
provides an analysis and case studies of four states currently employing or implementing the data matching 
methodology. 
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II. Local Program Reporting Pilot 



The main goal of the local program reporting was to assess the costs and difficulties in requiring 
local programs to conduct student follow-up survey methods to collect NRS outcome measures. Another 
important goal was for local programs to evaluate the usability and accuracy of all proposed measures, 
especially the educational functioning levels. The local program reporting methodology is likely to be the 
methodology that will be used to collect the NRS follow-up measures, at least initially, since the state-level 
survey cannot collect information on individual program performance required by the Workforce 
Investment Act It is also not likely that data matching will be not possible in most states for several 
years. Thus the local reporting pilot was the largest and most important pilot test conducted. 

In the spring of 1998, NRS staff asked state directors of adult education to volunteer and to 
recommend local programs in their states to participate in the pilot We selected eight local programs from 
the states that volunteered to represent all regions of the country and to include different size programs, 
provider types and instructional approaches. The states and local programs that participated in the pilot 
were: 



• Alabama: Northeast Alabama Area Adult Education Program, DeKalb County Vocational 
Technical School, Rainsville 

• Arizona: Gary Tang Adult Education Center, Glendale 

• California: Overfelt Adult School, San Jose 

• Iowa: Eastern Iowa Community College, Career Assistance Center, Davenport 

• New York: Monroe #I BOCES, Fairport 

• Ohio: Queen City Vocational Center, Cincinnati 

• Pennsylvania: Lackawanna Junior College, Towanda 

• Tennessee: Knox County Schools, Knoxville 

To participate in the pilot, we asked the local programs to collect all of the proposed NRS 
measures, except the family and community measures, for a three-month period, beginning about mid- 
September 1998 and continuing through early January 1999. During this period local programs were asked 
to use the new educational functioning level descriptors. In addition, the local programs conducted a 
follow-up phone survey of learners who left the program six months prior to the time of the survey. The 
survey sample was limited to 250 learners and programs were permitted to survey only one of their 
instructional sites or program components in their program. The survey collected the proposed NRS 
employment, further education and training and credential measures. Appendix A contains the local 
follow-up survey. 

With the exception of the California and Tennessee sites, all of the pn-ograms sampled all of their 
adult basic education students who had attended at a single site. The California site conducted the survey 
only on ESL students, and used a simplified version of the survey that was translated into Spanish, 
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Vietnamese, and Chinese. The Tennessee site conducted the survey of a sample of students who had 
been mandated to attend due to welfare requirements. 

Before the pilot test began, two representatives from each local program, and a state 
representative were oriented to the NRS and given training on the pilot test methodology. During the two- 
day training, each program was given a guide which contained the sampling procedure, the survey, and 
procedures on how to contact learners and conduct the follow-up. In addition, the training covered issues 
such as data collection, and effective use of data for program improvement purposes. 

At the end of the pilot test period, a debriefing meeting was held with at least one representative 
from each local program. Representatives from the local programs recounted their experiences and 
provided feedback on the follow-up. 

Pilot Test Procedures 

Alabama 

Alabama attempted 716 phone calls and completed 140 surveys (20% of phone calls resulted in a 
completed survey) during the months of October, November, and December 1998. The original sample 
included 188 students sampled randomly using the guidelines supplied in the pilot procedures manual. Most 
calls were attempted between 9:00 a.m. and 4:00 p.m., though a significant portion were completed as late 
at 7:00 p.m. Callers generally had good contact information for their students; the vast majority of calls 
that did not result in completed surveys were attributable to students being unavailable at that particular 
time. 



Arizona 

Arizona’s surveys and calling logs from October and most of November were lost due to 
unforeseeable circumstances (a car theft) before they could be returned. During the last two weeks of 
November and the first week of December, callers attempted 88 phone calls. This figure includes few 
students who were called more than once. The 88 calls resulted in 13 completed surveys (15% of phone 
calls resulted in completed surveys). Approximately one -third (31%) of the calls reached phone numbers 
that were disconnected, not current, or simply wrong for the students in the sample. In another 24% of 
the calls, the student was not available or a message was left for the student. Most of Arizona’s calls 
were completed between 2:00 p.m. and 4:00 p.m. 

California 

California’s sample contained a large number of ESL students whose native language was 
Chinese, Vietnamese or Spanish. The program conducted approximately 620 phone calls to compete 136 
total surveys in all four languages (22% of the phone calls resulted in completed surveys). Most calls 
were conducted either in the morning hours or after 6:00 p.m. The program translated the survey 
instrument into these two languages and administered the translated forms to those students who were not 
comfortable completing the original English version. 
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Iowa 



In Iowa, one caller attempted to contact the 1 13 students in the sample. Each student was called 
only once. These 1 13 phone calls resulted in 12 completed surveys (1 1% of the phone calls resulted in 
completed surveys). Iowa had significant problems with phones that were disconnected, wrong numbers, 
and not reaching anybody at the contact number. Most phone calls were completed between 4:00 p.m. 
and 8:00 p.m., though phone calls from early in the pilot test were made between 10:00 a.m. and 3:00 p.m. 

HenYotk 

New York’s data had not been received in time for inclusion in this report. 

Ohio 

Ohio’s sample included 536 students who had withdravm from the program during March, April, 
and May 1998, sampled randomly. Two instructional assistants (I As) called these students during 
November and December 1998. The lAs conducted an estimated 600 phone calls to complete 150 
surveys (25% of the calls made resulted in completed surveys). In over two-thirds of their calls, they 
reached a wrong number or busy line, received no answer, or left a message. Five percent of the sample 
refused to participate in the interview. Few members of the sample were called more than once. Most 
calls occurred from 3:00 p.m. to 7:00 p.m. weekday evenings. 

Pennsylvania 

Pennsylvania used three staff members to complete 107 phone calls to reach the 68 ABE and 
ESL students included in its sample. The 107 calls resulted in 37 completed surveys (35% of the calls 
resulted in completed surveys). Callers tended to have valid contact information, failing to reach students 
because they were not available or simply leaving messages. Calls were made at various times during 
business hours. 

Tennessee 

In Tennessee, seven employees attempted 748 calls during October, November, and December 
1998 to reach the 200 students included in their random sample. The 748 calls resulted in a total of 1 12 
completed surveys (15% of the calls resulted in completed surveys). Callers had difficulty completing 
interviews due to incorrect contact infimiation, unretumed messages, and the inability to reach anybody at 
the listed phone number. Most calls were made during the middle of the day (from 10:30 a.m. to 3:00 
p.m.) and in the early evening (from 5:00 p.m. to 7:30 p.m.). 

Pilot Test Findings 

Response Rates and Cost 

We received data from seven out of the eight local programs involved in the pilot test. The seven 
programs combined made a total of over 2,000 phone calls, and completed interviews for approximately 
400 learners. The overall interview completion rate was 23 percent, while the interview completion rate 
for the individual programs ranged from 1 1 to 35 percent. The sample size (where available), response 
rates, and call completion rates for each local program are shown in Figure 1 below. 
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Figure 1. Response and Call Completion Rates 



state 


Number In 
Sample 


Number of 
Surveys 
Completed 


Response 

Rate 


Number of 
Calls Made 


Call 

Completion 

Rate 


Alabama 


188 


140 


74% 


716 


20% 


Arizona 


n/ff 


13 


n/a 


88 


15% 


California 


n/gp 


136 


n/a 


620 


22% 


Iowa 


113 


12 


11% 


113 


11% 


New York* 


n/a 


n/a 


n/a 


n/a 


n/a 


Ohio 


536 


150 


28% 


600 


25%' 


Pennsylvania 


68 


37 


54% 


107 


35% 


Tennessee 


200 


112 


56% 


748 


15% 



^Arizona and California were unable to provide accurate sample sizes. 

^Estimated 

Calculation based on estimated denorninator 

isiew York did not return vaSd data regarding their calling program. 

Pilot test participants indicated that costs of conducting the follow-up survey over the three-month 
period ranged from $500 to over $1500. The largest categories of expense was staff time. At the 
debriefing, the largest concern expressed by the program representatives was the cost and availability of 
staff to conduct the follow-up surveys. Many of the programs are staffed by part-time and volunteer 
instructors and program had difficulty getting them to volunteer for a task for which they would not be 
compensated. It was also pointed out that in some locations, instructors are unionized, preventing the 
programs from even asking them to assist with the surveys. Alabama returned the only itemized expense 
report, which included time logs for four staff members and a catalog of supplies consumed in the survey 
process. Figure 2 shows their estimated expense categories. 



Figure 2. Alabama's Expense Categories 



Expense Category 


Quantity 


Secretary labor 


45.5 hours 


Program Coordinator labor 


11.5 hours 


Teacher labor (2 teachers) 


41 hours 


Photocopies (surveys and enrollment forms) 


600 


Long Distance Phone Calls and Faxes 


13caIIsb 



^Estimated 

^No time or cost was given for individual calls 



Pilot programs suggested that money (either state or Federal) be allocated specifically for WIA 
compliance, or the local programs may not choose or be able to spend the money for NRS activities. 

Improving the Survey and Procedures 

Some pilot participants thought the survey procedures could be simplified and offered suggestions 
for improving its design and administration. First, procedures should include a mail option for learners who 
do not have a valid telephone number and to otherwise increase response rates. Second, the programs 
also stated a desire for specific guidance on the number of learners to sample. Third, the NRS should 
have a mechanism for measuring “outcomes” for students still enrolled in the program. Fourth, it was 
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suggested that program staff should have input into when and how to conduct the survey and should then 
develop a survey plan. 

There were also a number of concerns regarding who and when to survey. The participants felt 
uncomfortable with the point in time at which students were surveyed. Among their suggestions were 
only surveying students who reached a certain threshold for contact hours and changing the cutoff point 
for including students in the sample (perhaps to the end of the year, rather than surveying students who 
had been separated for six months). 

Assessment of Educational Functioning Level Descriptors 

Local programs in the pilot also reviewed and assessed the proposed changes to the educational 
functioning levels. Several themes emerged from the comments provided by each of the programs. First, 
there was general agreement that the educational functioning levels worked well for ABE students. Most 
programs had little or no trouble converting from the current Federal levels to the new NRS levels. There 
was some sentiment that the levels were not sensitive enough to show progress at the lowest levels, 
especially for ESL students. It was noted, however, that this is also true of the current system. The 
participants also pointed out that the intake process was essential to providing accurate functioning level 
and for conducting the follow-up surveys. Overall, however, the programs reported general satisfaction 
with the functional level measures, even though they have different requirements for post-testing their 
students, summarized below. 

• Tennessee programs must show 1.5 grade level progression on the TABE after every 60 
hours of contact. 

• California programs test their students every 100 hours using CASAS. 

• Pennsylvania programs test their students every 50 hours using the TABE. 

• Arizona program test students every quarter with the TABE. 

• New York programs were required to test their students every 50 hours using the TABE. 

• Iowa programs test every 60 hours (on average) using either the TABE or ABLE. 

• Alabama programs test their students at the teacher’s discretion, usually with the TABE. 

• Ohio programs have no hourly requirement for testing, though some programs test quarterly or 
at the end of the semester. 

Assessment of the Methodology for the NRS 

Advantages 

The local program follow up presents several advantages as a model for the NRS. Its primary 
advantage is that the sampling universe is at the program level, rather than at the state level. The resulting 
data provides states with accountability data for each individual program within that state. States can use 
this information to make determinations regarding funding for local programs. Furthermore, local 
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programs can use these data for program improvement. Another primary advantage is the relatively high 
response and call completion rates compared to those in the state follow up pilot test (described below). It 
seems that local programs are better able to collect reach students than are states. 

Barriers and Disadvantages 

There are, however several disadvantages to this methodology. The costs related to conducting 
surveys are high, especially for local programs not used to conducting this type of data collection. As 
indicated by pilot test participants, the method is extremely time-consuming and few programs currently 
have the resources to conduct telephone surveys. Second, with each program conducting its own survey 
collection, there is relatively little centralized oversight over the data collection procedures. Combined with 
the high cost of conducting the survey, local programs may submit poor quality data. Finally, the 
population surveyed proved to be very difficult to contact, with many students being totally unreachable. 

In a number of instances students were reluctant to participate in the survey. Alabama, in fact, reported 
that potential students were dissuaded from even enrolling in their program when they heard that someone 
from the Adult School would be calling them to collect information on them once they left the program. 
The California site found that the ESL population is very difficult to contact and that language and cultural 
issues create further difficulties in contacting these students. 
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III. state Follow-up Survey 



The state follow-up survey pilot was designed to test the feasibility of using a periodic state -level 
survey of students to collect the longer-term student outcome measures related to employment, further 
education and training and credentials. Rather than having each local program in a state conduct a survey 
on a sample of its students, the state survey methodology has state staff conduct a survey on a statewide 
sample of adult education students. This methodology appears attractive since it alleviates from local 
programs the burden of collecting follow-up data and also reduces the overall cost of conducting the 
survey. 



The disadvantage of this method is that it does not allow data to be tied back to an individual local 
program, since the sample for the survey is drawn from the universe of all students in the state. When the 
pilot was planned, this shortcoming was not considered to be a major problem. However, just before the 
pilot test was to begin, the Workforce investment Act was passed, which requires states to evaluate the 
performance of individual programs on the outcome measures. This requirement means that the state 
level survey cannot be used as a methodology for the NRS. Despite this issue, we decided to continue 
with plans to conduct the pilot as a means of obtaining further information on the feasibility of conducting 
follow-up surveys to obtain data. 

As with the local program follow-up pilot test, the main goal of this pilot was to assess the costs 
and barriers to implementing this data collection approach, which could then be compared to the local 
program methodology. The states of Idaho, Maryland and Rhode Island volunteered to participate in the 
state follow-up pilot. These states used a telephone -administered interview conducted with a sample of 
former adult education students sampled throughout the state who had left the program six months earlier. 
Unlike the local survey, the state survey gathered information on the community and family outcome 
measures in addition to the employment, further education and credential measures. It was thus 
considerably longer than the local survey. 

The NRS staff supplied sampling guidelines, the survey instrument itself, and necessary technical 
assistance to the participating states. Appendix A contains a copy of the survey. 

Though the NRS provided the general tools and methods for collecting the follow-up data, the 
states were responsible for developing the specific procedures for conducting the follow-up survey. Each 
state implemented the methodology in a different way. 

• Idaho turned responsibility for the survey over to the University of Idaho. 

• Maryland hired a private contractor that had worked for the state on previous data collection 
efforts. 

• The state director of adult education administered Rhode Island’s surveys with assistance 
from a welfare client in a work experience program. 

All three of these systems adequately met the requirements of the NRS, and are likely similar to those that 
would be devised in a full-scale implementation of this methodology. 
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Pilot Test Procedures 



Idaho 

The state office of adult education in Idaho relied on the University of Idaho to conduct the 
surveys. The Center for Educational Research and Public Service in the College of Education had 
previously coordinated and conducted similar data collection efforts under funding provided by the state for 
a needs assessment for adult literacy. The office of adult education capitalized on this relationship to have 
the University conduct the NRS survey using funding provided for the previous needs assessment. 

The Office of Adult Education instructed all programs in the state to submit to the University lists 
of students who had exited the programs during May or June 1998. These lists included approximately 
2,000 names. The University then drew a sample of 1,741 students for inclusion in the survey. The state 
director of adult education reported that one or more programs did not submit all of the information 
necessary to contact the learners. These programs were instructed to submit all of the necessary 
information, a process that delayed the surveying of their students by several weeks. The University used 
four student callers to complete 207 interviews during mid-December 1998 and early- to mid-February 
1999. 



Maryland 

The Maryland State Department of Education (MSDE) contracted with the University of 
Maryland at College Park to conduct the interviews. MSDE drew a random sample of 580 learners by 
computer. During the sampling process, MSDE collected Social Security numbers for all students and 
sorted the sample by the programs they attended. They then sent this list to each of the programs to 
collect contact information for each learner (this information is not kept in databases at the state level). 

The individual programs returned contact information to MSDE, which passed social security numbers and 
contact information (but no names) to the contractor. 

The contractor in turn used three callers to complete the interviews over the course of the pilot 
test. Callers were instructed to make no more than five attempts to complete an interview. A total of 104 
interviews were completed. The contractor was paid $4,198 to conduct the survey. 

Rhode Island 

The director of adult education for Rhode Island took personal responsibility for administering the 
survey. He enlisted the help of a “workfare” participant who was in community work experience as a 
condition of receiving public assistance. Using manual procedures suggested in the NRS manual, they 
drew a sample of approximately 140 students from throughout the state. These students had been enrolled 
in every type of program in the state, including volunteer organizations; community based organizations 
(CBOs); and learning centers. Approximately 40 interviews were completed. The callers had great 
difficulty reaching learners to complete the survey. The calls themselves were completed during business 
hours, primarily from 10:00 a.m. to about 3:30 p.m. The director reported that the process was very 
frustrating since the majority of calls made did not result in completed surveys. 
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Pilot Test Findings 



We evaluated the feasibility of the state survey methodology test for collecting NRS measures by 
assessing the response rates, costs, and difficulties states encountered in conducting the survey. 

Response Rates and Cost 

Callers at the University of Idaho made 1,763 individual phone calls and completed 207 interviews. 
Thus, approximately 12% of phone calls resulted in a completed interview. Out of the 1,741 students 
deemed eligible for the survey, 207 were reached and completed interviews, for an overall response rate 
of approximately 12%. 

Callers at the University of Maryland reported making nearly 1,700 calls to complete 104 
interviews. Thus, approximately 6% of phone calls resulted in a completed interview. The overall 
response rate in Maryland was approximately 18% (104 of the 580 students in the sample were actually 
interviewed). 

While Rhode Island did not keep a precise log of the number of phone calls made, the state 
director estimated that about 10-12% of all attempted calls resulted in completed interviews. The overall 
response rate for this state was approximately 29% (40 of the 140 students in the sample were actually 
interviewed). These response rates were comparable to those obtained in the local program reporting 
pilot. Figure 3 summarizes these findings. 



Figure 3. Call Completion and Response Rates 



state 


Number In 
Sample 


Number of 
Surveys 
Completed 


Response 

Rate 


Number of 
Calls Made 


Call 

Completion 

Rate 


Idaho 


1,741 


207 


11.9% 


1,763 


11.7% 


Maivland 


580 


104 


17.9% 


1,700 


6%b 


Rhode Island 


140 


40 


28.6% 


40CM8O 


10-12%b 



aEstimated. 

^Calculation based on estirnated denominator 



The cost for obtaining a complete survey varied from $40.37 in Maryland to $132.04 in Idaho and 
is shown in Figure 4. In both of these states, contractors conducted the surveys under very different 
compensation arrangements. In Idaho, the survey was done as a part of a pre-existing contract between 
the state office of adult education and the University. There were, therefore no new fimds allocated for 
the pilot test activities. In Maryland, by contrast, the contractor received funds allocated specifically for 
the pilot test. Rhode Island did not provide an estimate of the cost of conducting the survey. Comparisons 
of the cost of a contractor cannot, therefore, be compared to the cost of program staff at this time. 
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Figure 4. Costs per Completed Interview 



state 


Number of Surveys 
Completed 


Total Cost 


Cost per Completed 
Survey 


Idaho 


207 


$27333 


$13204 


Maryland 


104 


$4,198 


$40.37 


Rhode Island 


40 


Not Available 


Not Aval table 



Idaho provided greater detail on the cost of the survey. The state provided no additional funding 
specifically for the follow-up survey. The University estimated the cost of administering the survey at just 
over $27,000, as shown in Figure 5. 



Figure 5. Expense Breakdown for Idaho 



Expenditures 


Cost 


Personnel 


Principal Investigator 


11350 


Research Assistant 


1300 


Research Ass^t 


2,500 


Fringe @28.5 


4,354 


Irregular Help 


2,850 


Fringe @13.1 


373 


Total Personnel 


18,400 


Total Fringe 


4,727 


SUBTOTAL 


23,127 


Operating Expenses 


Phone 


1,638 


Copies 


497 


Postage 


327 


Misc. 


244 


SUBTOTAL 


2,706 


Travel 


Boise (Strips) 


1,500 


SUBTOTAL 


1,500 




TOTAL 


27,333 



Difficulties in Completing Surveys 

As with the local program reporting pilot, the response rates for the survey were very low. States 
reported reasons for non-response similar to those reported by the local programs. The majority of Rhode 
Island’s phone calls were not completed due to incorrect contact information (disconnected or incorrect 
phone numbers). When the callers did reach someone other than the learner or an answering machine at 
the phone number listed, they left a message. In all but a few cases, the messages were not returned. 

The director of Rhode Island’s program also reported that the completed surveys mostly contained 
information from ASE students. Collecting information from ABE and ESL students proved to be much 
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more difficult. The director felt that the survey itself was too long and that the verbiage was difficult for 
those without strong English skills. 

The contractors from both Idaho and Maryland reported having specific difficulties with incorrect 
contact information. At least one local program in Idaho submitted incomplete information for all of its 
students. This delayed calls to students from that program for nearly a month. The principal investigator 
in Idaho reported that a small but significant amount of learners contacted unexpectedly terminated the 
interview halfway through because of its length. The she, along with the principal investigator from 
Maryland, reported following the pilot test guidelines and listing large proportions of their samples as non- 
respondents after failing to reach them after 3 to 4 attempts. 

Assessment of the Methodology for the NRS 

Advantages 

The state follow up presents several advantages over the local program follow up methodology. 
First, it is significantly less costly than the local survey. By administering the survey to a state sample of 
students, as opposed to individual student samples for each local program, the overall cost of data 
collection is reduced. Idaho was, in fact, able to conduct the survey without any new expenditure by 
taking advantage of a pre-existing arrangement with the University of Idaho. Maryland also contracted 
with the State University to conduct the survey at relatively low cost. Finally, Rhode Island used only two 
staff members to conduct its interviews. 

A second advantage to the state option is that it alleviates the burden of data collection from each 
local program and centralizes it at the state level or to a state contractor. By using contractors, as in 
Maryland and Idaho, the state was relieved of the burden of data collection. There the state Universities 
actually did the data collection and took on the burdens of providing staff and conducting calls. Local 
programs would rarely be able to afford to hire a third party contractor. 

Barriers and Disadvantages 

The state survey option is greatly facilitated by a state -level database with student data, including 
student contact information. None of the three states in this pilot had an adequate database at the state 
level. In both Maryland and Idaho, local programs had to be contacted to obtain the contact information 
and other pertinent information for the survey. In Rhode Island, manual procedures had to be used. This 
lack of information created substantial delays in the conduct of the survey. The use of this methodology 
would require states to develop an individual level student record database at the state level, which 
apparently would be a significant challenge for most states. 

The biggest drawback to this methodology, however, is that the state survey does not allow 
outcomes to be identified at the individual program level. The data can be used to assess the performance 
of the state program as a whole on the outcome measures, but as currently designed, cannot be used for 
assessing the performance of individual programs. This drawback is fatal for using methodology in the 
NRS, given the current requirements of the Workforce Investment Act. Section 231(e)(2) of the Act 
requires that the states assess local program performance on the core outcome measures, which include 
the follow-up measures used in the pilot. The lack of information on individual programs would prevent 
states from complying with the data requirements of the WIA. Thus, this methodology cannot assist in 
meeting the accountability requirements of the law. 
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IV. Data Matching 



Data matching refers to the procedures where two or more agencies pool and share data on a 
common group of participants to match students to outcomes. The data consist of individual student 
records collected by each of the agencies that can be linked through a common identifier, typically a Social 
Security number. Matching the pooled data using the common identifier produces a new individual student 
record or an aggregated data report containing data from one or more of the additional agencies. Each 
agency can use the new, pooled data records or reports to understand the impact of their program on 
participants and to obtain data to meet their reporting and accountability requirements. 

Data matching methods are particularly well suited for studying outcomes that occur some time 
after program participation. For example, wage record information systems have been used to study the 
outcomes of vocational education and employment programs. The WIA requires job training programs 
funded under Title I to use a data matching methodology to obtain the required employment outcomes. 

For the NRS, this methodology holds considerable promise for studying the follow-up employment 
measures. Among the reasons making it attractive are its cost relative to surveys, decreased data 
collection burdens, and increased data validity. 

The first major advantage of data matching is that it is significantly less costly than the survey 
methodologies. The cost of conducting a survey — drawing a sample, training interviewers, making phone 
calls — are replaced with the cost of combined, cleaning and analyzing the data. Further, this cost can be 
divided among the participating agencies. 

The second major advantage of data matching is reduced data collection burden. At the local 
program level, staff no longer needs to conduct survey procedures. Local programs would collect only the 
demographic, participation and educational fimctioning level information. Finally, matched data are likely 
to be more valid than those collected through surveys, which are self-reported data. Some former 
students may not be tmthfiil about whether they obtained a job, for example. The wage or unemployment 
record data base, however, would reveal whether students are working. 

The Data Matching Pilot 

The local program and state survey pilots assessed the cost and data quality resulting from using 
these methodologies by having programs and states conduct the data collection on a limited scale. With 
data matching, it was not possible to ask a state or program to begin using matching procedmes, since 
such procedmes need data systems and interagency collaboration that require considerable time to put in 
place. To identily the issues in establishing and using data matching as an approach for the NRS, we 
studied four states that are using this methodology. 

The four states investigated in the NRS pilot test have the most advanced systems relevant to 
adult education programs, although they are in different stages of development. Florida and Illinois have 
already created their systems and are conducting analyses and using shared data for accountability. 
California and Oregon have only begun implementation of shared data arrangements within the last two 
years. In examining these systems we focused on: 

• The development of the systems, including why they were developed and the problems 
encountered in establishing them; 
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• The measures included in each system; 

• How each system operates to perform the data matching and reporting; 

• Management of the system and cost, including start-up and maintenance costs; and 

• Training and technical assistance need to implement the systems. 

Development of the Systems 

The development of collaborative data systems occurred because of mandates from the state 
legislatures of Florida, California, and Oregon, which differed in their specific reasoning for requiring these 
systems. Generally, however, the legislatures wanted uniform accountability and planning systems for 
education and training programs. In Florida the mandate originally covered only the K- 12 system but was 
broadened to include adult education. Illinois’s system evolved from a project originally funded by the 
National Governor’s Association to develop integrated state management information systems for 
workforce development programs. Illinois was one of six states awarded a grant to develop this system 
and work continued to complete and implement the system after the grant ended. 

The states intended their integrated systems to include data from all agencies that deliver 
employment and training services. Heads of the involved agencies debgated responsibility for developing 
and implementing the system to mid-level officials, with the power to act on their behalf as they made 
decisions affecting the project. The involvement of mid-level career officials helped the development and 
implementation of the system to continue without regard to any individual agency director, who often 
changed. In the four states, this continuity allowed the staff to become personally involved in the project 
and remain dedicated to its completion, continuing their work in spite of frequent changes among agency 
leadership. 

In each of the states, the need for employment-related outcomes were a primary force in the 
systems’ development. For this reason, the states’ Departments of Labor or their equivalent were major 
participants in the initial development. Other agencies and departments that deliver training and 
employment services, or had an interest in the services provided to their clients, were also heavily involved 
in the development process. Among these are agencies and departments that oversee: state budgets; 
commerce and economic development; community colleges; corrections; education; employment and 
unemployment programs; health; other higher education providers (e.g., post secondary commissions, state 
university systems, and boards of higher education); the Job Partnership Training Act program (now Title 
1 under the Workforce Investment Act); labor; vocational rehabilitation; and the welfare agency. Figure 6 
presents a summary of the agencies involved in each state. 
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Figure 6. Agencies invoived in Deveioping 
and impiementing Coiiaborative Data Systems 



Agency/Department 


California 


Florida 


Illinois 


Oregon 


Budget Bureau 




✓ 






Corrirr^erceEcoii C)e^«toprT^ 




/ 


✓ 




Community Colleges 


✓ 


✓ 


✓ 


✓ 


Confections 


✓ 


✓ 


✓ 


✓ 


Education 


✓ 


✓ 


✓ 


✓ 


Empbyment/Unemployment Services 


✓ 


✓ 


✓ 


✓ 


Health 


✓ 








Other Higher Education 


✓ 


✓ 


✓' 




JPTA 




✓ 


✓ 


✓ 


lEtor 


✓ 


✓ 




✓ 


Vocational RehabiFitation 


✓ 




✓ 




Weiiae 


✓ 


✓ 


✓ 


✓ 



1 Date is onV collected on program entiy, not o)^ 



Barriers to Deveiopment 

The most significant barriers to the development of these systems have been confidentiality issues 
around Social Security numbers, development of measure definitions common across agencies, and clients 
whose program participation status changes are not reflected as they cross state lines for employment 
reasons. 



Data matching arrangements require participant Social Security numbers as a common identifier. 
However, there are legal barriers to sharing of Social Security numbers across agencies due to 
confidentiality issues. Each state has resolved this problem somewhat differently. In Oregon, each 
agency is permitted only to use the aggregate data within its own agency and cannot identify individual 
participants. Consequently, reports are only provided in aggregate form. Likewise in Florida, only 
aggregate reporting is permitted, though data can be aggregated for the entire workforce development 
system, for the main components or agencies of the system, or for individual state and local workforce 
development activities. In Illinois, state agencies may share data and with Social Security numbers for 
research purposes only and the state has ruled that data sharing for accountability is permitted as research, 
though this is likewise carried out with aggregate data. In California, program participants must sign a 
release form for their information to be used in the common database. Many participants do not sign the 
release, creating significant information gaps. For example, about 50 percent of target participants in Los 
Angeles County do not sign the release. 

For matched data to be meaningful, the shared data items should have a common definition or 
meaning. In Florida, it has been easier for community colleges and local education agencies (LEAs) to 
work with non-educational agencies than to work with each other due to the lack of common definitions 
and historically different data collection systems. Even though the legislature has mandated a common 
system for the education agencies, differences remain, and the state continues to work on them. 

Illinois provides a specific example. The shared data base is intended to include only program 
“completers,” but the term is defined differently for adult education, job training participants, and higher 
education students. In adult education, a student who advances an educational functioning level is 
considered a completer. For job training programs, a completer is someone how has been employed for 90 
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days; for higher education, a completer must have received an associate degree and/or a completion 
certificate. It was not possible to develop a common definition that all agencies agreed upon. Instead, the 
agreed upon solution has been the measurement creation of “accountability cohorts” in which each agpncy 
uses its mandated or traditional definition of a completer. 

Oregon similarly lacked the definitions necessary to describe its workforce development clients. 
The basic skills curriculum in the state is housed in the state community college system. This system, 
however, had no way of recording data on its students enrolled in this curriculum. In order to adopt a set 
of workable definitions for these clients, the state adopted an entirely new data system, TOPS. This 
system allows the state to track basic skills students in the community colleges and to include that data 
with other information reported by the college. 

A final problem which has not been resolved is that fact that Unemployment Insurance data, 
which tracks wages and employment, is only valid in the state in which the person first applied. 
Employment data will not, therefore, be available for clients that take jobs in neighboring states. The 
extent of this problem is not known, although Illinois estimated that they lose about four percent of 
participants due to their moving out of the state. 

Data Collected and Reported in the Systems 

The four states in this pilot test collect two major types of data. The first type, demographic and 
descriptive information, gives programs the ability to accurately describe their client base to their 
constituent audiences. Outcome measures and other performance indicators, the second type of data 
collected, provide the states with mechanisms for program accountability. 

Among the demographic and descriptive measures most commonly collected are: years of 
education, highest level completed, ethnicity, reasons for enrollment, educational functioning level (usually 
based on standardized test scores), employment status, participation in JTPA or welfare programs, and 
special needs. These measures can be aggregated to provide a detailed portrayal of program participants, 
allowing judgements about target population access to various programs. 

There is great similarity in terms of the outcome measures and performance indicators that can be 
reported. 

• Illinois can report on 35 different measures collected by participating agencies, including core 
education, welfare, employment, and income measures. Customer satisfaction measures 
were defined, but never implemented. An adult education report, for example, might indicate 
that a client received a GED or high school diploma, got a job, was employed 12 and 24 
months later, participated in continuing education, and left welfare. Information for these 
measures is available through Unemployment Insurance (UI) records, monthly welfare 
benefits databases, the shared higher education database, and community college enrollment 
data. 

• In California, outcome measures include job placement, wages and earnings, length of 
employment and employment rates are currently reported. Learning gains, GED attainment 
and employer satisfaction may be added later. 
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• Oregon will report on job placement, job retention, wages and earnings, welfare reduction, 
welfare recidivism, return on investment, attahment of a GED or credentials, and completion 
of work/community learning experience. Educational measures will soon be added including 
completion of educational fimctional levels, attainment of workforce readiness (through 
attainment of specific skills), and skill gains in reading, writing, thinking, speaking, and listening; 
and English language acquisition. 

• In Florida, performance indicators focus specifically on the workforce development system. 
Data collection focuses on welfare cessation, wages earned, and employment status, 
continuation and retention in higher education, and customer satisfaction. In education, data 
are collected by districts and include types of program provided, enrollments, completions, 
placement as determined by teachers, and other performance indicators. 

Among the four states, only Oregon included all adult education students in the shared data 
system. California’s system includes only ABE and ESL students who have employment as a primary or 
secondary goal, and adults enrolled in Regional Occupational Centers (ROC) and in adult vocational 
education. In Florida, students participating in GED and other adult education programs are included. 
Illinois’s shared data system includes all adult education students. 

System Operation 

Under a data matching system, each participating agency collects a common core of demographic 
and descriptive information on their participants, dates of program participation, a common identification 
number (Social Security number) and the outcome measures specific to its program. All measures that 
are shared among the agencies need to have common definitions if the resulting analyses and reports are 
to be meaningful across agencies. 

Under the central data processing or data warehousing model, each agency submits its individual 
client records containing the data to be shared to a central source (either a contractor or in-house agency). 
This central agency combines the information into a single data pool and eliminates record duplication 
using Social Security numbers. This data pool is then available to the individual agencies, which can 
request specific tables and reports. The reports are usually in aggregate form at the state, program and 
site level although individual data reporting can be produced. Local program providers can also request 
reports through their agencies. Figure 7 shows this model of data matching. 
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Figure 7 




Example of Shared Interagency Data Base— Data Warehouse 



Under a second, decentralized or data harvesting model of data matching, each agency maintains 
its own data records and each separate agency requests matches from the agency with the needed data. 
To match with an outside agency, the requesting agency sends records containing Social Security numbers 
and other data needed for the analysis to another agency, along with the format of the data tables needed. 
The other agency makes the matches and reports the data in the requested format. For example, to obtain 
the GED test results of students, the state could send the Social Security numbers of students who had a 
goal of passing the GED tests, along with demographic and program information, to the state agency that 
does GED testing. The testing agency would match the records to produce a report on the number and 
characteristics of students who passed the GED tests. The state could then use this information in its 
annual NRS reporting. 

For both types of data matching, incorrect or missing Social Security numbers affects the 
availability of data. This problem can be substantial if students refose to provide their Social Security 
number or provide incorrect numbers. Legal barriers to collecting Social Security numbers also pose a 
significant barrier to this methodology. Another serious problem affecting data analyses with data 
matching is the time lag from the end of the reporting period until when the data are available. It often 
takes two or more quarters for all of the data to be available in agencies. Among the four states, the time 
lag ranges from one quarter to a year. For example, if a student leaves the program in February, entered 
employment is measured in the next quarter (April-June). However, if the time lag is two quarters, that 
student’s data could not be matched until the first quarter of the following calendar year. 

Management and Cost 

The four states have various management mechanisms to organize the data and control access 
and use of the data. External contractors, committees or agencies provide oversight and management. 

The four states also report vastly different costs for developing and maintaining the system. 

In Illinois, an outside contractor. Northern Illinois University (NIU), receives the data collected by 
the participating agencies. An interagency committee, composed of representatives from each agency 
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contributing to the data system, provides oversight NIU. State staff estimated that less than $100,000 was 
used to develop the MIS data management process. Costs to develop the data matching process were 
minimized the state had already automated UI records and software that enabled data matching, providing 
a blueprint for the new MIS. Maintenance costs are estimated to be approximately $50,000 per year. 

When Oregon’s basic skills system began, the Office of Community College Services (OCCS) 
was responsible for maintaining the database. The development costs for the initial assessment 
instruments and data collection system were about $80,000. OCCS is now responsible for maintaining the 
educational and employment records on clients in its system. A new data system, TOPS, was developed 
for adult education to participate in the shared information system. Costs for implementing the TOPS 
system included $250,000 in local infrastmcture improvements and an additional $85,000 in state 
technology support expenses. Currently, OCCS provides assessment training to all agencies, while the 
Employment Department with interagency co-fimding funds the maintenance of the MIS itself, which is 
estimated at $750,000. 

The Florida Education and Training Placement Information Program (FETPIP) was created by an 
act of the Florida legislature in 1988. FETPIP serves as the central data warehouse for information from 
workforce development agencies, compiling individual student or participant files submitted electronically 
by participating organizations. The current FETPIP annual budget is approximately $600,000, of which 
$400,000 comes from general state funding, $140,000 comes from the Department of Labor and Economic 
Security, and $60,000 comes from federal Perking Act resources. These funds pay for nine professional- 
and one support- staff members. 

In California, the data report will be provided to the Special Committee on Performance Based 
Accountability (PBA), which is under the SJTCC. From its inception, the data management system was 
funded by a small grant from JTPA. Currently, the state is considering a proposal to tax the agencies 
submitting data at the rate of $6 per client. There is concern that this high cost will be prohibitive to small 
providers, who will then be unable to meet reporting requirements and lose funding. 

Training and Technicai Assistance for System impiementation 

The development and implementation of a shared data system requires a considerable investment 
in training and technical assistance at the local level to ensure that the data provided are valid and reliable. 
Teachers, for example, typically feel overburdened by data collection requirements and all four states 
reported resistant to and lack of understanding of the reasons for data collecting and reporting. For these 
reasons, three of the states have substantially invested in training. 

• In Illinois, three full-time staff members provide technical assistance on the state adult 
education data system to local adult education programs. There have been over 100 training 
sessions, and more than $1 million of Section 353 funds have been spent to implement the 
system. 

• In Oregon, more than sbc years have elapsed since the decision to move from a system 
describing only student needs and demographics to one that focuses on outcomes. Training 
teachers has also proved costly; 50% of professional development time is devoted to this 
issue, coordinated by a half-time technical assistance person. 
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• Although in California, the shared system is not fully yet imptemented in adult education, 
training on using data systems is a high priority. CASAS and the California Staff 
Development Institute have been providing this training on an ongoing basis. 

There has been little training of adult education providers in Florida, as adult education has only a recent 
and limited involvement in the integrated system. 

Implications for the NRS 

The use of data matching arrangements is an attractive approach toward collecting the follow-up 
measures in the NRS. However, the experiences of California, Florida, Illinois, and Oregon in 
implementing this approach have shown that it takes time to develop and operate the system. While 
Illinois was able to get its system going within two years, Oregon required about five years and California 
and Florida’s systems do not fiilly involve adult education after several years. Also crucial to 
implementation is an interagency planning process with individuals committed to seeing the system 
development. This process is successful when political concerns are kept out of the planning and 
development. 

Another essential ingredient is for each agency to have an automated, individual participant record 
system. The lack of such a system for adult education in Oregon delayed development and this problem 
remains a barrier to adult education’s involvement in Florida. It is not necessary, however, that each 
agency use the same record system or software, only that the software used by each agency can produce 
information in a common format to allow data matching. 

Beyond these basic planning and infrastructure needs, the experience of the four states has 
identified three conceptual problems that need to be surmounted for to develop shared data arrangements: 
the need for common outcome and measure definitions, the need to address concerns about data 
confidentiality and the need for high quality training and technical assistance. 

The MIS must have common definitions for outcome measures for measures that will be shared. 
Agencies with jurisdiction over different types of programs (for example. Departments of Labor and 
Education) must provide data that are based on common understandings of the measures. Furthermore, 
agencies within a single department (for example, community colleges and LEAs) must also use common 
definitions. Care must be taken, however, to ensure that the definitions agreed upon maintain their fidelity 
with the mission of the program. If common agreement on common definitions cannot be reached, each 
agency must understand what the other definitions are and must be able to accommodate these 
differences in interpreting the data. For example, if “program completion” is a common data element, 
each agency must use the same definition or must have an understanding of what the other definitions are 
and interpret the data accordingly. 

The issue of confidentiality looms large over data matching procedures. Each of the four states 
had to resolve the prohibition about sharing Social Security numbers across agencies. In California, the 
state superintendent initially refused to allow the education department to participate because of this 
problem. The state now requires permission from students to share their records, and since many students 
refuse, there are significant gaps in the data. The other three states resolved this problem by defining the 
data matching as research and/or by allowing only aggregated reporting, so that individual students cannot 
be identified. 
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Many states have laws against not only sharing Social Security numbers, but against sharing any 
sharing educational records. These barriers must be resolved legally before data matching can become a 
widely used methodology for the MRS. 

Finally, a great deal of training and technical assistance at the local level is needed to develop a 
system that produces valid and reliable data. Training should be provided on measure definitions, data 
collection and reporting, and data use. Such training also produces buy-in to the whole data collection and 
analysis process and can help “convert” teachers, local statf, other stakeholders who might be skeptical 
about the usefulness of the system. The training can also provide local providers with an idea of how the 
data is used at the state level, as well as how they can use it to improve their program. , 
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California’s Shared Data System 

History and Development 



A 1996 state law requires the State Job Training Coordinating Council to develop and implement an 
outcome-based system to measure the performance of state- and federally-funded education and training 
programs that comprise the state workforce preparation system. The SJTCC is to report on individual 
programs and the system as a whole through a series of “report cards” that include these outcomes. 
Implementation began in program year 1997, but education did not become involved until 1998-1999 when 
issues about using Social Security numbers were resolved. 

Agencies Involved 



Adult Education Community Colleges Job Service 

Vocational Rehabilitation Vocational Education Unemployment Insurance 

Job Training Programs Department of Corrections Department of Social Services 

Employment and Training Panel 



Adult Education Population Included 



Only adult education students are specifically pursuing career enhancement or job training, which are adults 
enrolled in Regional Occupational Centers (ROC), adult vocational education and adults in ABE and ESL 
programs who have employment as a primary or secondary goal. 

Outcome Measures 



Job placement, wages and earnings, length of employment and employment rates are currently reported. 
Learning gains, GED attainment and employer satisfaction may be added later. 

Operation 

Each agency collects demographic information on clients, participation dates. Social Security Numbers and 
its own outcome measures. These data are sent to a third party contractor, who make the data matches and 
then sends aggregated reports back to each agency in the format it requests. There is about a one-year time 
lag from submission until data are available. 

Barriers to Development and Implementation 

The most significant barrier to implementation has been concern about whether agencies could require 
students to provide Social Security numbers, which prevented involvement of Department of Education 
programs. This issue was resolved by making the reporting of Social Security numbers voluntary. The cost 
of operating the system has also slowed implementation. 
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Management and Cost 



The SJTCC established the interagency Performance-Based Accountability (PBA) Committee to develop and 
oversee this system. The cost for the data matching is estimated at about $6 per client matched. 

Florida’s Shared Data System 



History and Development 

The Florida Education and Training Program (FETPIP), an interagency data collection system, was 
established in 1984, but remains in development for educational programming. Since 1993, participating 
agencies have been working to clarify definitions and data elements. FETPIP exists to obtain follow-up data 
on former workforce development program participants and provide this data to programs and organizations 
in which the participant received services. 

Agencies Involved 



Public Schools 
State Universities 
Department of Corrections 
Job Training Programs 

Adult Education Population Included 



Adult Education 
Community Colleges 

Selected Private Vocational Education Programs 
Department of Labor and Em ployment Security 
Department of Health and Rehabilitative Services 



Ail students who enroll in adult education are included in the system. 

Outcome Measures 

Government job status (including job classification and pay grade), wages, length of employment of clients, 
welfare participation or reduction, enrollment in other education programs, enlistment in the military, 
employer satisfaction with training of program participants. 

Operation 



Each agency collects its own outcome measures. Once a year, FETPIP uses Social Security numbers to 
match participant fiies electronically to administrative records or other state and Federal agencies. 
Employment records are updated quarterly. Customer satisfaction measures are collected through annual 
surveys of a sample of employers hiring program participants. 

Barriereto Development and Implementation 



The major remaining obstacle has been the development of a single data collection system for community 
colleges and LEAs. The lack of common definitions also remains problematic. Education programs have 
accepted the system, however, because of its link to annual funding. 



Management and Cost 



FETPIP receives and compiles all student and participant data submitted from participating workforce 
development agencies’ MISs. In 1984, the state allocated $60,000 for initial study and design plans, though 
the current annual budget is approximately $600,000. Of this, $400,000 comes from general state funding, 
$140,000 comes from the Department of Labor and Economic Security, and $60,000 comes from federal 
Perking Act resources. These funds pay for nine professional- and one support- staff members. 

Illinois’ Shared Data System 



History and Development 

Beginning in the summer of 1 994, Illinois was one of six states working with the National Governors’ 
Association (NGA) to develop a framework for managing and measuring performance across workforce 
development programs. The state Adult Education program worked in conjunction with representatives from 
employment, training, and welfare agencies to develop workforce performance measures. Most of the work 
in the NGA project involved identifying the common measures and developing appropriate measurement 
strategies. 

Agencies Involved 

Adult Education 
Job Training Programs 
.. Unemployment Insurance 
Community Colleges 

Adult Education Population Included 

All students who enroll in adult education are included in the system. 

Outcome Measures 

Educational measures include attainment of credentials (including certificates of mastery, high school 
diploma, high school equivalency certificate, postsecondary degree or certificate, skills credential and 
competency credential), learning gains, and matriculation to further education and training. Economic 
measures include entering employment (including subsidized and unsubsidized employment, employment 
with earnings constraints, and employment related to training), retaining current employment, wage at 
placement, earnings gains, average earnings over time, welfare reduction, and unemployment insurance 
savings. 

Operation 

Each agency collects demographic information on clients, participation dates. Social Security Numbers and 
its own outcome measures. These data are sent to a third party contractor, who makes the data matches 
and sends aggregated reports to each agency in a requested format. Individual level data are not available. 
Economic measures are extracted from the Unemployment Insurance database. 



Vocational Education 
Employment and Training Panel 
Veterans Affairs 
Department of Social Services 



Barriers to Development and Implementation 

There has been some difficulty in aligning some of the outcome measures and definitions. The term 
“completer," for example, is defined differently for education, JTPA clients, and higher education clients. In 
addition. Unemployment Insurance data is not available for clients that take jobs in neighboring states. 
Finally, the quarterly reporting of Unemployment Insurance data creates a lag for the economic measures 
that does not exist for education measures. 

Management and Cost 

It is estimated that less than $1 00,000 was used to develop the MIS data management process. Costs to 
develop the data matching process were minimized the state had already automated University of Illinois 
records, providing a blueprint for the new MIS. Maintenance costs are estimated to be $50,000 per year. 

Oregon’s Shared Data System 



History and Development 

Oregon' s education, job training and social service agencies began working together in 1987 to determine 
the characteristics of Oregon's welfare population in the wake of welfare reform. The state legislature wanted 
to be able to describe Oregon's long range goals and to develop a systems approach to measuring and 
meeting those needs. In 1992, the legislature required agencies to work together to develop a coordinated 
state plan and two years later asked for core measures for the workforce development system. Prior to 1 994, 
the system provided for client assessment only at intake. The mandates under which Oregon operated 
forced the development and implementation of the MIS to continue without regard to changing agency 
directors. 

Agencies Involved 

Office of Community College Services 
Employment Department 
Adult and Family Services 
Economic Development Agency 
Governor’s Office of Workforce & Educational Policy 

Adult Education Population Included 

All students who enroll In adult education are Included in the system. 

Outcome Measures 

Job placement, job retention, wages and earnings, welfare reduction, welfare recidivism, return on 
investment and attainment of a GED or credentials are reported. Educational measures will soon be added 
including completion of educational functional levels, workforce readiness (through attainment of specific 
skills), and completion of work/community learning experience. 



Office of Professional and Technical Education 
Department of Corrections 
Oregon Workforce Advisory Committee 
Job Training Partnership Act Administration 
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Operation 



Each agency collects demographic information on clients, participation dates, Social Security Numbers and 
its own outcome measures. When other agencies want information, they provide to the relevant agency the 
format of the report desired, student Social Security numbers and demographic or other data as appropriate 
for the report requested. The individual agency returns the report or tables in aggregate form. Tables can be 
by program or site, but not by individual students, due to confidentiality rules. There is about a nine-month 
time lag from data submission by the individual agencies until data are available for reports. 

Barriere to Development and Implementation 

The entire process has taken considerable time as data systems needed to be developed and the 
development of the system required continual reaffirmation and support. The process has been facilitated by 
the fact that adult education and employment training programs are both under the same agency— the Office 
of Community College Services. 

Management and Cost 

Each agency maintains its own data records and pays for its own reports. The Office of Community College 
Services oversees the education records and the Oregon Workforce Cabinet and Workforce Advisory 
Committee oversee the Shared Information System housed at the Oregon Employment Department. The 
development costs for the system for adult education were approximately $400,000. Including local program 
infrastructure and annual maintenance and reporting, the cost of the system is approximately $750,000. 
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